Teleoperated surgical system with surgeon skill level based instrument control

ABSTRACT

A teleoperated surgical system is provided comprising: a first robotic surgical instrument; an image capture; a user display; a user input command device coupled to receive user input commands to control movement of the first robotic surgical instrument; and a movement controller coupled to scale a rate of movement of the first robotic surgical instrument, based at least in part upon a surgical skill level at using the first robotic surgical instrument of the user providing the received user input commands, from a rate of movement indicated by the user input commands received at the user input command device.

CLAIM OF PRIORITY

This application claims the benefit of priority to U.S. PatentApplication No. 62/421/072, filed on Nov. 11, 2016, which is herebyincorporated by reference herein in its entirety.

BACKGROUND 1. Field of Invention

Inventive aspects are associated with medical devices used duringsurgery. More specifically, aspects are associated with surgicalinstrument use in accordance with surgeon skill level.

2. Art

Surgeons typically undertake extensive study before performing asurgical procedure. Traditionally, surgeons were limited to the study ofgeneric anatomical models, such as photographs or drawings. Morerecently, various pre-operative diagnostic procedures (e.g., x-ray, CT,MRI, etc.) have made patient-specific anatomical information available.

In some cases, it is desirable to make additional, relevant anatomic andsurgical procedure information available to a surgeon. In one aspect, itis desirable to provide a surgeon planning an operation on a particularpatient with a surgical site video recording of an earlier surgicalprocedure performed on the particular patient. In another aspect, it isdesirable to provide a surgeon with one or more surgical videorecordings of surgical procedures on other patients that are similar tothe surgical procedure planned for a particular patient. In one aspect,it is desirable to provide such information to a surgeon prior to thesurgeon undertaking a particular surgical procedure. And in anotheraspect, it may be desirable to provide this information to a surgeonintraoperatively.

In one aspect, it is desirable to configure a video database thatincludes intraoperative surgical site video recordings of variousprocedures undergone by various patients. In one aspect, it is desirableto configure a medical device capable of video recording to furtherinclude an input that enables a surgeon using the medical device tohighlight and annotate the video recording in real time as it is beingrecorded. In one aspect, it is desirable to configure a computer-basedpattern matching algorithm to search through the individual records ofthe video database, identify relevant video records, and provide asurgeon with this relevant information for a particular surgicalprocedure.

SUMMARY

The following summary introduces certain aspects of the inventivesubject matter in order to provide a basic understanding. This summaryis not an extensive overview of the inventive subject matter, and it isnot intended to identify key or critical elements or to delineate thescope of the inventive subject matter. Although this summary containsinformation that is relevant to various aspects and embodiments of theinventive subject matter, its sole purpose is to present some aspectsand embodiments in a general form as a prelude to the more detaileddescription below.

A surgical method is provided for use with a teleoperated surgicalsystem that includes a robotic surgical instrument. An image capturedevice is orientable toward a surgical site for capturing images ofanatomical tissue and of robotic surgical instrument. A user display iscoupled to the image capture device to show to a user, the capturedimages of the anatomical tissue and of the robotic surgical instrument.A user input command device is coupled to receive user input commands tocontrol movement of the robotic surgical instrument. A movementcontroller circuit is coupled to receive the user input commands fromthe input command device. The movement controller circuit is configuredto control movement of the robotic surgical instrument in response tothe user input commands. The movement controller circuit is furtherconfigured to scale a rate of movement of the robotic surgicalinstrument, based at least in part upon a surgical skill level at usingthe robotic surgical instrument of the user providing the received userinput commands, from a rate of movement indicated by the user inputcommands received at the user input command device.

A method is provided to operate a teleoperated surgical system thatincludes a robotic surgical instrument manipulator. User input commandsare received from a user to control movement of a robotic surgicalinstrument mounted at the robotic surgical instrument manipulator. Anidentification determination is made of a robotic surgical instrumentmounted at the robotic surgical instrument manipulator during thereceiving the user input commands. A rate of movement of the roboticsurgical instrument is scaled, based at least in part upon a skill levelof the user at, use of the identified surgical instrument, from a rateof movement indicated by the user input commands.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a minimally invasive teleoperated surgicalsystem.

FIG. 2 is a perspective view of a surgeon's console.

FIG. 3 is a perspective view of an electronics cart.

FIG. 4 is a diagrammatic illustration of a teleoperated surgical system.

FIG. 5 is a perspective view of a patient-side cart.

FIG. 6 is an elevation view of a surgical instrument.

FIG. 7 is a perspective view of an instrument manipulator.

FIG. 8 is a diagrammatic illustration of a surgical planning tool.

FIG. 9 is a flow diagram of a method of using a surgical planning tool.

FIG. 10 is an illustrative drawing representing storage atlas in acomputer readable storage device in accordance with some embodiments.

FIG. 11 is an illustrative drawing representing an example of the ninthinformation structure included within the atlas in the storage device,which associates recorded video information from an individual surgerywith corresponding surgical instrument actuator state information andsurgeon eye movement information in accordance with some embodiments.

FIGS. 12A-12C are illustrative drawings showing an example surgicalinstrument and an actuator assembly in which the surgical instrument isshown in three different example operational states in accordance withsonic embodiments.

FIG. 13 is an illustrative drawing representing an example tenthinformation structure of the atlas stored in the computer readablestorage device 1004 that corresponds to an example surgical procedure tobe performed by an example surgeon.

FIG. 14 is an illustrative flow diagram representing configuration of aprocessor to scale kinematic translation of user-to-instrument movementaccording to a surgical instrument safety actuation state based at leastin part upon surgeon skill level information in accordance with someembodiments.

DETAILED DESCRIPTION

This description and the accompanying drawings that illustrate inventiveaspects, embodiments, implementations, or applications should not betaken as limiting the claims define the protected invention. Variousmechanical, compositional, structural, electrical, and operationalchanges may be made without departing from the scope of this descriptionand the claims. In some instances, well-known circuits, structures, ortechniques have not been shown or described in detail in order not toobscure the invention. Like numbers in two or more figures represent thesame or similar elements.

Elements described in detail with reference to one embodiment,implementation, or application may, whenever practical, be included inother embodiments, implementations, or applications in which they arenot specifically shown or described. For example, if an element isdescribed in detail with reference to one embodiment and is notdescribed with reference to a second embodiment, the element maynevertheless be claimed as included in the second embodiment. Thus, toavoid unnecessary repetition in the following description, one or moreelements shown and described in association with one embodiment,implementation, or application may be incorporated into otherembodiments, implementations, or aspects unless specifically describedotherwise, unless the one or more elements would make an embodiment orimplementation non-functional, or unless two or more of the elementsprovide conflicting functions.

Aspects of the invention are described primarily in terms of animplementation using a da Vinci® Surgical System (specifically, a ModelIS 4000, marketed as the da Vinci® Xi™ HD™ Surgical System),commercialized by intuitive Surgical, Inc. of Sunnyvale, Calif.Knowledgeable persons will understand, however, that inventive aspectsdisclosed herein may be embodied and implemented in various ways,including robotic and, if applicable, non-robotic embodiments andimplementations. Implementations on da Vinci® Surgical Systems (e.g.,the Model IS4000 da Vinci® Xi™ Surgical System, the Model IS3000 daVinci Si® Surgical System) are merely exemplary and are not to beconsidered as limiting the scope of the inventive aspects disclosedherein.

In accordance with various aspects, the present disclosure describes asurgical planning tool that includes a medical device configured tovideo record the performance of surgical procedures. The videorecordings can be embedded with various metadata, e.g., highlights madeby a medical person. Additionally, the video recordings can be taggedwith various metadata, e.g., text annotations describing certain subjectmatter of the video, the identity of the patient to whom the videorecording corresponds, biographical or medical information about thepatient, and the like. In one aspect, tagged metadata is embedded in thevideo recordings.

In accordance with further aspects, the present disclosure describes ateleoperated medical device that includes a surgical instrument used toperform at least one surgical activity during a surgical procedure.Different stages of a surgical activities may require different surgicalskill levels. In some embodiments, a surgical level in a surgicalactivity may be determined based at least in part upon a comparison ofthe surgeon's performance level of the surgical activity with theperformance levels of other surgeons in the activity. A surgery mayinvolve use of different surgical instruments during different portionsof a surgical procedure. Each surgical instrument used during a surgeryis controlled by one or more surgical instrument actuators operable inmultiple actuator states. Which surgical instrument is in use duringdifferent portions of a surgery is tracked. In some embodiments, anactuator state of an actuator controlling a surgical instrument that isin use is tracked during surgical procedures. In some embodiments,surgeon eye movement also is tracked using a camera to determinedirection of surgeon gaze during the surgery. In some embodiments, aninformation structure in a computer readable storage device associatessurgical instrument in use and surgical instrument actuator states withsurgical guidance information for presentation to a surgeon in responseto a surgical instrument's use to perform the at least one surgicalactivity. In some embodiments, the surgical guidance information that ispresented to a surgeon is determined based at least in part upon thesurgeon's surgical skill level. In some embodiments, an informationstructure in a computer readable storage device associates at least oneof a surgical instrument use during surgery or its specific actuatorstates during the performance of the at least one surgical activity withsafety transition information for use to cause the surgical instrumentactuator to transition to an actuator safety state of operation thatmatches a surgeon's skill level. In some embodiments, the surgicalinstrument actuator safety state of operation is determined based atleast in part upon a surgeon's skill level.

In a teleoperated surgical system, different instruments may be used atdifferent stages of a surgical procedure. Moreover, the same instrumentmay be used in different actuator states at different stages of asurgical procedure. As used herein, the term actuator state refers to amechanical disposition of a surgical instrument as determined by anactuator, such as a motor, in response to input commands received from asurgeon or other surgical team member.

The video recordings and information structures that associate surgicalinstrument's use or specific actuator states with surgical guidance oractuator safety state information can be archived on an electronicmedical record database implemented locally or on a cloud data storageservice. The video recordings can be made available to interested healthcare providers. The information structures can be made available for usewith the teleoperated medical device to provide surgical guidance and tocontrol surgical instrument actuator state during performance of atleast one surgical activity during performance of a surgical procedure.

Health care providers can search the medical device database based uponsurgeon skill level for videos and information structure relationshipsof interest using the metadata tags described above. Additionally, inone aspect, the surgical planning tool includes a computer-based patternmatching and analysis algorithm. In one aspect, the pattern-matchingalgorithm culls through the videos stored on the electronic medicalrecord database to identify correlations between visual characteristicsin the video recordings and associated metadata tags made by medicalpersons. The surgical planning tool can apply these correlations tonewly encountered anatomy, and thereby assist medical persons performinga procedure in making determinations about patient anatomy, preferredsurgical approaches, disease states, potential complications, etc. Inanother aspect, the pattern matching algorithm culls through videosstored on the electronic medical record database to identifycorrelations between visual characteristics in the video recordings andpatient health record information to identify patient anatomicalcharacteristics that correlate with surgeon skill level information. Thesurgical planning tool can apply these correlations between anatomy andsurgeon skill level records to a current patient's anatomy and healthrecords, and thereby assist medical persons planning and performing asurgical procedure involving the current patient.

Minimally Invasive Teleoperated Surgical System

Referring now to the drawings, in which like reference numeralsrepresent like parts throughout the several views, FIG. 1 is a plan viewof a minimally invasive teleoperated surgical system 10, typically usedfor performing a minimally invasive diagnostic or surgical procedure ona patient 12 who is lying on an operating table 14. The system includesa surgeon's console 16 for use by a surgeon 18 during the procedure. Oneor more assistants 20 may also participate in the procedure. Theminimally invasive teleoperated surgical system 10 further includes apatient-side cart 22 and an electronics cart 24. The patient-side cart22 can manipulate at least one removably coupled surgical instrument 26through a minimally invasive incision in the body of the patient 12while the surgeon 18 views the surgical site through a user displaywithin the surgeon's console 16. An image of the surgical site can beobtained by an image capture device such as a stereoscopic endoscope 28,which can be manipulated by the patient-side cart 22 to orient theendoscope 28 so as to capture images of patient anatomical structuresand one or more surgical instruments at a surgical site. Computerprocessors located on the electronics cart 24 can be used to process theimages of the surgical site for subsequent display to the surgeon 18through a stereoscopic display at the surgeon's console 16. The numberof surgical instruments 26 used at one time will generally depend on thediagnostic or surgical procedure and the space constraints within theoperating room among other factors. If it is generally necessary tochange one or more of the surgical instruments 26 being used during aprocedure, an assistant 20 can remove the surgical instrument 26 fromthe patient-side cart 22, and replace it with another surgicalinstrument 26 from a tray 30 in the operating room.

FIG. 2 is a perspective view of the surgeon's console 16. The surgeon'sconsole 16 includes a user display that includes a left eye display 32and a right eye display 34 for presenting the surgeon 18 with acoordinated stereoscopic view of the surgical site that enables depthperception. The console 16 further includes an input command device thatincludes one or more manual control inputs that include hand grips 36,38. One or more surgical instruments installed for use on thepatient-side cart 22 (shown in FIG. 1) move in response to surgeon 18 ′smanipulation of the one or more control inputs 36, 38. The controlinputs 36, 38 can provide the same mechanical degrees of freedom astheir associated surgical instruments 26 (shown in FIG. 1) to providethe surgeon 18 with telepresence, or the perception that the controlinputs 36, 38 are integral with the instruments 26 so that the surgeonhas a strong sense of directly controlling the instruments 26. To thisend, position, force, and tactile feedback sensors (not shown) may beemployed to transmit position, three, and tactile sensations from thesurgical instruments 26 back to the surgeon's hands through the controlinputs 36, 38.

The surgeon's console 16 is usually located in the same room as thepatient so that the surgeon can directly monitor the procedure, bephysically present if necessary, and speak to a patient-side assistantdirectly rather than over the telephone or other communication medium.But, the surgeon can be located in a different room, a completelydifferent building, or other remote location from the patient allowingfor remote surgical procedures.

FIG. 3 is a perspective view of the electronics cart 24. The electronicscart 24 can be coupled with the endoscope 28 and includes a computerprocessor to process captured images for subsequent display, such as toa surgeon on the surgeon's console 16, or on another suitable displaylocated locally and/or remotely. For example, if a stereoscopicendoscope is used, a computer processor on electronics cart 24 canprocess the captured images to present to the surgeon at the left andright eye displays 32, 34, coordinated stereo images of the surgicalsite. Such coordination can include alignment between the opposingimages and can include adjusting the stereo working distance of thestereoscopic endoscope. As another example, image processing can includethe use of previously determined camera calibration parameters tocompensate for imaging errors of the image capture device, such asoptical aberrations. Optionally, equipment in electronics cart may beintegrated into the surgeon's console or the patient-side cart, or itmay be distributed in various other locations in the operating room.

FIG. 4 diagrammatically illustrates a teleoperated surgical system 50(such as the minimally invasive teleoperated surgical system 10 of FIG.1). A surgeon's console 52 (such as surgeon's console 16 in FIG. 1) canbe used by a surgeon to control a patient-side cart 54 (such aspatent-side cart 22 in FIG. 1) during a minimally invasive procedure.The patient-side cart 54 can use an imaging device, such as astereoscopic endoscope, to capture images of a surgical site and outputthe captured images to a computer processor located on an electronicscart 56 (such as the electronics cart 24 in FIG. 1). The computerprocessor typically includes one or more data processing boards purposedfor executing computer readable code stored in a non-volatile memorydevice of the computer processor. In one aspect, the computer processorcan process the captured images in a variety of ways prior to anysubsequent display. For example, the computer processor can overlay thecaptured images with a virtual control interface prior to displaying thecombined images to the surgeon via the surgeon's console 52.

Additionally, or in the alternative, the captured images can undergoimage processing by a computer processor located outside of electronicscart 56. In one aspect, teleoperated surgical system 50 includes anoptional computer processor 58 (as indicated by dashed line), whichincludes one or more central processing units (CPUs) similar to thecomputer processor located on electronics cart 56, and patient-side cart54 outputs the captured images to computer processor 58 for imageprocessing prior to display on the surgeon's console 52. In anotheraspect, captured images first undergo image processing by the computerprocessor on electronics cart 56 and then undergo additional imageprocessing by computer processor 58 prior to display on the surgeon'sconsole 52. Teleoperated surgical system 50 can include an optionaldisplay 60, as indicated by dashed line. Display 60 is coupled with thecomputer processor located on the electronics cart 56 and with computerprocessor 58, and captured images processed by these computer processorscan be displayed on display 60 in addition to being displayed on adisplay of the surgeon's console 52.

Moreover, the control inputs 36, 38 are coupled to receive user inputcommands to control movement of one or more surgical instruments at thesurgical site. The processor 58 acts as a kinematic movement controllercircuit that is coupled to receive the user input commands from thecontrol inputs 36, 38. The processor 58 translates user input in theform of physical movement of the control inputs 36, 38 to controlsignals to control motors to control corresponding movement of one ormore surgical instruments to a movement controller within the patientside cart 54 to impart corresponding movement to an endoscope or to oneor more surgical instruments. The translation of user input movementimparted by a user's hand motions upon control inputs 36, 38 tocorresponding instrument movement imparted by motors coupled to thesurgical instruments involves kinematic movement translation, whichtypically involves scaling of distances such that an instrument may bemoved by only a small fraction of the distance that a control inputs 36or 38 is moved to impart a user command to cause the instrumentmovement. In other words, user input movement imparted to control inputs36, 38 in user space is translated to corresponding smaller scalemovements in instrument space at the surgical site. An example ofkinematic movement translation in a teleoperated surgical system isdescribed in U.S. Pat. No. 6,424,885.

FIG. 5 is a perspective view of a patient-side cart 500 of a minimallyinvasive teleoperated surgical system, in accordance with embodiments ofthe present invention. The patient-side cart 500 includes one or moresupport assemblies 510. A surgical instrument manipulator 512 is mountedat the end of each support assembly 510. Additionally, each supportassembly 510 can optionally include one or more unpowered, lockablesetup joints that are used to position the attached surgical instrumentmanipulator 512 with reference to the patient for surgery. As depicted,the patient-side cart 500 rests on the floor. In other embodiments,operative portions of the patient-side cart can be mounted to a wall, tothe ceiling, to the operating table 526 that also supports the patient'sbody 522, or to other operating room equipment. Further, while thepatient-side cart 500 is shown as including four surgical instrumentmanipulators 512, more or fewer surgical instrument manipulators 512 maybe used.

A functional minimally invasive teleoperated surgical system willgenerally include a vision system portion that enables a user of theteleoperated surgical system to view the surgical site from outside thepatient's body 522. The vision system typically includes a camerainstrument 528 for capturing video images and one or more video displaysfor displaying the captured video images. In some surgical systemconfigurations, the camera instrument 528 includes optics that transferthe images from a distal end of the camera instrument 528 to one or moreimaging sensors (e.g., CCD or CMOS sensors) outside of the patient'sbody 522. Alternatively, the imaging sensor(s) can be positioned at thedistal end of the camera instrument 528, and the signals produced by thesensor(s) can be transmitted along a lead or wirelessly for processingand display on the one or more video displays. One example of a videodisplay is the stereoscopic display on the surgeon's console in surgicalsystems commercialized by intuitive Surgical, Inc., Sunnyvale, Calif.

Referring to FIG. 5, mounted to each surgical instrument manipulator 512is a surgical instrument 520 that operates at a surgical site within thepatient's body 522. Each surgical instrument manipulator 512 can beprovided in a variety of forms that allow the associated surgicalinstrument to move with one or more mechanical degrees of freedom (e.g.,all six Cartesian degrees of freedom, five or fewer Cartesian degrees offreedom, etc.). Typically, mechanical or control constraints restricteach manipulator 512 to move its associated surgical instrument around acenter of motion on the instrument that stays stationary with referenceto the patient, and this center of motion is typically located at theposition where the instrument enters the body.

In one aspect, surgical instruments 520 are controlled throughcomputer-assisted teleoperation. A functional minimally invasiveteleoperated surgical system includes a control input that receivesinputs from a user of the teleoperated surgical system (e.g., a surgeonor other medical person). The control input is in communication with oneor more computer-controlled teleoperated actuators, such as one or moremotors to which surgical instrument 520 is coupled. In this manner, thesurgical instillment 520 moves in response to a medical person'smovements of the control input. In one aspect, one or more controlinputs are included in a surgeon's console such as surgeon's console 16shown at FIG. 2. A surgeon can manipulate control inputs 36 of surgeon'sconsole 16 to operate teleoperated actuators of patient-side cart 500.The forces generated by the teleoperated actuators are transferred viadrivetrain mechanisms, which transmit the forces from the teleoperatedactuators to the surgical instrument 520.

Referring to FIG. 5, in one aspect, a surgical instrument 520 and acannula 524 are removably coupled to manipulator 512, with the surgicalinstrument 520 inserted through the cannula 524. One or moreteleoperated actuators of the manipulator 512 move the surgicalinstrument 512 as a whole. The manipulator 512 further includes aninstrument carriage 530. The surgical instrument 520 is detachablyconnected to the instrument carriage 530. In one aspect, the instrumentcarriage 530 houses one or more teleoperated actuators inside thatprovide a number of controller motions that the surgical instrument 520translates into a variety of movements of an end effector on thesurgical instrument 520. Thus the teleoperated actuators in theinstrument carriage 530 move only one or more components of the surgicalinstrument 520 rather than the instrument as a whole. Inputs to controleither the instrument as a whole or the instrument's components are suchthat the input provided by a surgeon or other medical person to thecontrol input (a “master” command) is translated into a correspondingaction by the surgical instrument (a “slave” response).

In an alternate embodiment, instrument carriage 530 does not houseteleoperated actuators. Teleoperated actuators that enable the varietyof movements of the end effector of the surgical instrument 520 arehoused in a location remote from the instrument carriage 530, e.g.,elsewhere on patient-side cart 500. A cable-based force transmissionmechanism or the like is used to transfer the motions of each of theremotely located teleoperated actuators to a correspondinginstrument-interfacing actuator output located on instrument carriage530. In some embodiments, the surgical instrument 520 is mechanicallycoupled to a first actuator, which controls a first motion of thesurgical instrument such as longitudinal (z-axis) rotation. The surgicalinstrument 520 is mechanically coupled to a second actuator, whichcontrols second motion of the surgical instrument such astwo-dimensional (x, y) motion. The surgical instrument 520 ismechanically coupled to a third actuator, which controls third motion ofthe surgical instrument such as opening and closing or a jaws endeffector.

FIG. 6 is a side view of a surgical instrument 520, which includes adistal portion 650 and a proximal control mechanism 640 coupled by anelongate tube 610 having an elongate tube centerline axis 611. Thesurgical instrument 520 is configured to be inserted into a patient'sbody and is used to carry out surgical or diagnostic procedures. Thedistal portion 650 of the surgical instrument 520 can provide any of avariety of end effectors 654, such as the forceps shown, a needledriver, a cautery device, a cutting tool, an imaging device (e.g., anendoscope or ultrasound probe), or the like. The surgical end effector654 can include a functional mechanical degree of freedom, such as jawsthat open or close, or a knife that translates along a path. In theembodiment shown, the end effector 654 is coupled to the elongate tube610 by a wrist 652 that allows the end effector to be oriented relativeto the elongate tube centerline axis 611. Surgical instrument 520 canalso contain stored (e.g., on a semiconductor memory inside theinstrument) information, which may be permanent or may he updatable by asurgical system configured to operate the surgical instrument 520.Accordingly, the surgical system may provide for either one-way ortwo-way information communication between the surgical instrument 520and one or more components of the surgical system.

FIG. 7 is a perspective view of surgical instrument manipulator 512.Instrument manipulator 512 is shown with no surgical instrumentinstalled. Instrument manipulator 512 includes an instrument carriage530 to which a surgical instrument (e.g., surgical instrument 520) canbe detachably connected. Instrument carriage 530 houses a plurality ofteleoperated actuators. Each teleoperated actuator includes an actuatoroutput 705. When a surgical instrument is installed onto instrumentmanipulator 512, one or more instrument inputs (not shown) of aninstrument proximal control mechanism (e.g., proximal control mechanism640 at FIG. 6) are mechanically coupled with corresponding actuatoroutputs 705. In one aspect, this mechanical coupling is direct, withactuator outputs 705 directly contacting corresponding instrumentinputs. In another aspect, this mechanical coupling occurs through anintermediate interface, such as a component of a drape configured toprovide a sterile barrier between the instrument manipulator 512 anassociated surgical instrument.

In one aspect, movement of one or more instrument inputs bycorresponding teleoperated actuators results in a movement of a surgicalinstrument mechanical degree of freedom. For example, in one aspect, thesurgical instrument installed on instrument manipulator 512 is surgicalinstrument 520, shown at FIG. 6. Referring to FIG. 6, in one aspect,movement of one or more instrument inputs of proximal control mechanism640 by corresponding teleoperated actuators rotates elongate tube 610(and the attached wrist 652 and end effector 654) relative to theproximal control mechanism 640 about elongate tube centerline axis 611.In another aspect, movement of one or more instrument inputs bycorresponding teleoperated actuators results in a movement of wrist 652,orienting the end effector 654 relative to the elongate tube centerlineaxis 611. In another aspect, movement of one or more instrument inputsby corresponding teleoperated actuators results in a movement of one ormore moveable elements of the end effector 654 (e.g., a jaw member, aknife member, de.). Accordingly, various mechanical degrees of freedomof a surgical instrument installed onto an instrument manipulator 512can be moved by operation of the teleoperated actuators of instrumentcarriage 530.

Annotating a Recorded Video

FIG. 8 shows a schematic diagram of an exemplary surgical planning tool800. In one aspect, surgical planning tool 800 includes a teleoperatedsurgical system 850 in data communication with an electronic medicaldevice record database 830. Teleoperated surgical system 850 shown hereis similar to teleoperated surgical system 850 shown at FIG. 4. In oneaspect, electronic medical record database 830 includes the medicalrecords of patients that have undergone treatment at a particularhospital. Database 830 can be implemented on a server located on-site atthe hospital. The medical record entries contained in the database 830can be accessed from hospital computers through an intranet network.Alternatively, database 830 can be implemented on a remote serverlocated off-site from the hospital, e.g., using one of a number of clouddata storage services. In this case, medical record entries of database830 are stored on the cloud server, and can be accessed by a computerwith internet access.

In one aspect, a surgical procedure is performed on a first patientusing teleoperated surgical system 850. An imaging device associatedwith teleoperated surgical system 850 captures images of the surgicalsite and displays the captured images as frames of a video on a displayof surgeon's console 52. In one aspect, a medical person at surgeon'sconsole 52 highlights or annotates certain patient anatomy shown in thedisplayed video using an input device of surgeon's console 52. Anexample of such an input device is control input 36 shown at FIG. 2,which is coupled to a cursor that operates in conjunction with a graphicuser interface overlaid onto the displayed video. The graphic userinterface can include a QWERTY keyboard, a pointing device such as amouse and an interactive screen display, a touch-screen display, orother means for data or text entry. Accordingly, the medical person canhighlight certain tissue of interest in the displayed image or enter atext annotation.

In one aspect, the surgical site video is additionally displayed on adisplay located on electronics cart 56. In one aspect, the display ofelectronics cart is a touch-screen user interface usable by a medicalperson to highlight and annotate certain portions of patient anatomyshown on an image that is displayed for viewing on the display on theelectronics cart. A user, by touching portions of patient anatomydisplayed on the touch-screen user interface, can highlight portions ofthe displayed image. Additionally, a graphic interface including aQWERTY keyboard can be overlaid on the displayed image. A user can usethe QWERTY keyboard to enter text annotations.

In one aspect, the surgical site video captured by the imaging deviceassociated with teleoperated surgical system 850 is recorded by theteleoperated surgical system 850, and stored on database 830, inaddition to being displayed in real time or near real time to a user.Highlights and/or annotations associated with the recorded video thatwere made by the user can also be stored on database 830. In one aspect,the highlights made by the user are embedded with the recorded videoprior to its storage on database 830. At a later time, the recordedvideo can be retrieved for viewing. In one aspect, a viewer of therecorded video can select whether the highlights are displayed orsuppressed from view. Similarly, annotations associated with therecorded video can also be stored on database 830. In one aspect, theannotations made by the user are used to tag the recorded video, and canbe used to provide as a means of identifying the subject mattercontained in the recorded video. For example, one annotation maydescribe conditions of a certain disease state. This annotation is usedto tag the recorded video. At a later time, a person desiring to viewrecorded procedures concerning this disease state can locate the videousing a key word search.

Retrieval of Stored Video

In some cases, it is desirable for a medical person to be able to viewvideo recordings of past surgical procedures performed on a givenpatient. In one aspect, a patient who previously underwent a firstsurgical procedure to treat a medical condition subsequently requires asecond surgical procedure to treat recurrence of the same medicalcondition or to treat anatomy located nearby to the surgical site of thefirst surgical procedure. In one aspect, the surgical site events of thefirst surgical procedure were captured in a surgical site videorecording, and the video recording was archived in database 830 as partof the patient's electronic medical records. Prior to performing thesecond surgical procedure on the patient, a medical person can perform asearch of database 830 to locate the video recording of the patient'searlier surgical procedure.

In some cases, it is desirable for a medical person planning to performa surgical procedure on a patient to be able to view video recordings ofsimilar surgical procedures performed on persons having certaincharacteristics similar to the patient. In one aspect, surgical sitevideo recordings of surgical procedures can be tagged with metadatainformation such as the patient's age, gender, body mass index, geneticinformation, type of procedure the patient underwent, etc., before eachvideo recording is archived in database 830. In one aspect, the metadatainformation used to tag a video recording is automatically retrievedfrom a patient's then-existing medical records, and then used to tag thevideo recording before the video recording is archived in database 830.Accordingly, prior to performing a medical procedure on a patient, amedical person can search database 830 for video recordings of similarprocedures performed on patients sharing certain characteristics incommon with the patient. For example, if the medical person is planningto use teleoperated surgical system 850 to perform a prostatectomy on a65-year-old male patient with an elevated body mass index using, themedical person can search database 830 for surgical site videorecordings of prostatectomies performed using teleoperated surgicalsystem 850 on other males of similar age and having similarly elevatedbody mass index.

In one aspect, a video recording of a surgical procedure is communicatedby database 830 to an optional personal computer 820 (as indicated bydashed line), and made available for viewing by a medical person whoplans to perform a surgical procedure. Additionally, or in thealternative, the video recording of the earlier surgical procedure canbe communicated by database 830 to teleoperated surgical system 850, andmade available for viewing preoperatively or intraoperatively. In oneaspect, the video recording is displayed by teleoperated surgical system850 on a display located on surgeon's console 52. In another aspect, thevideo recording of the first surgical procedure is displayed on adisplay located on electronics cart 56.

Cloud-Based Video Database

In one aspect, database 830 is implemented on a remote server using acloud data storage service and is accessible by multiple health careproviders. Referring to FIG. 8, as shown by dashed line, surgicalplanning tool 800 optionally includes teleoperated surgical system 850(as indicated by dashed line) and personal computer 840 (as indicated bydashed line). In one aspect, teleoperated surgical system 850 is similarto teleoperated surgical system 850 and personal computer 840 is similarto personal computer 820, except that teleoperated surgical system 850and personal computer 820 are located at a first health care providerand teleoperated surgical system 850 and personal computer 840 arelocated at a second health care provider. In one aspect, a first patientrequires surgical treatment of a medical condition, and undergoes asurgical procedure using teleoperated surgical system 850 at the firsthealth care provider. A video recording of the surgical procedure isarchived on database 830. At a later time, a second patient requiressurgical treatment of the same medical condition, and plans to receivesurgical treatment using teleoperated surgical system 850 at the secondhealth care provider. Prior to performing the surgical procedure on thesecond patient, a medical person accesses database 830 through a secureinternet connection and searches database 830 for surgical site videorecordings of similar procedures. In one aspect, the medical persontreating the second patient is able to retrieve from database 830 thevideo recording of first patient's surgical procedure, without acquiringknowledge of the identity of the first patient. In this manner, theprivacy of the first patient is maintained. In one aspect, the videorecording of the first patient's surgical procedure includes highlightsand/or annotations made by the medical person who treated the firstpatient.

Computer Based Pattern Matching and Analysis

Surgical planning tool 800 can includes a pattern matching and analysisalgorithm implemented in the form of computer executable code. In oneaspect, the pattern matching and analysis algorithm is stored in anon-volatile memory device of surgical planning tool 800, and isconfigured to analyze the video recordings archived in database 830. Asdiscussed previously, each of the video recordings archived in database830 can be tagged and/or embedded with certain metadata information.This metadata information can include patient information such aspatient age, gender, and other information describing the patient'shealth or medical history. Additionally, as discussed previously, themetadata information can include highlights or annotations made by amedical person. In one aspect, these highlights and annotations areembedded with the video recording and archived together with the videoin database 830.

In one aspect, pattern matching and analysis algorithm includes an imageanalysis component that identifies patterns in shapes and colors thatare shared amongst multiple video recordings stored on database 830. Thepattern matching and analysis algorithm then reviews the tagged metadataassociated with this subset of video recordings to determine whether anywords or phrases are frequently associated with videos within thissubset. These analyses performed by pattern matching and analysisalgorithm can be used to assist medical persons in making determinationsabout patient anatomy, preferred surgical approaches, disease states,potential complications, etc.

A Method of Using a Surgical Planning Tool

FIG. 9 shows a method 900 of using a surgical planning tool. In oneaspect, the surgical planning tool is similar to surgical planning tool800 at FIG. 8. At 910, a fact or characteristic describing a medicalpatient, e.g., a medical condition suffered by a patient, is received bya medical device, Medical device can receive this fact or circumstancevia a user interface located on a teleoperated surgical systemteleoperated surgical system 10 at FIG. 1 or teleoperated surgicalsystem 50 at FIG. 4), or alternatively, through a personal computersimilar to personal computer 820 at FIG. 2. At 920, the medical deviceuses the fact or characteristic received at 910 to retrieve at least onerelevant video recording of a surgical procedure from a medical devicedatabase. At 930, the medical device uses the video recordings todetermine surgical planning information. In one aspect, the surgicalplanning information includes the types of instruments used in therecorded procedure. At 940, the medical device displays to a user thesurgical planning information determined at 930.

A Method of Using Surgical Skill Level to Guide a Surgical Procedure

Chart 1 identifies several example distinct core set of surgicalinstrument skills have been identified that are useful during ateleoperated surgical procedure in accordance with Assessment of RoboticConsole Skills (ARCS) criteria.

CHART 1 Surgical Skill Bimanual wrist manipulation Camera Control Masterclutching to manage hand position Use of a third instrument armActivating an energy source Appropriate depth perception Awareness offorces applied by instruments

In some embodiments, surgical skill level for a category are rated asnovice, intermediate and experienced. A surgeon's skill level may varyfrom one skill category to the next. The skill assessment scale isgenerally applicable to any multiport robotically assisted surgicalprocedure, regardless of surgical specialty.

FIG. 10 is an illustrative drawing representing storage atlas in acomputer readable storage device 1004 in accordance with someembodiments. The storage atlas 1002 includes first informationstructures 1006 that indicates instances of previously performedsurgical procedures. A second information structures 1008 that indicatesteleoperated surgical instrument actuation states during the previouslyperformed surgical procedures. A third information structure 1010associates a surgical procedure with surgical activities during thesurgical procedure. A fourth information structures 1012 associates asurgeon with the surgeon's surgical skill levels. A fifth informationstructure 1014 associates surgical procedures with surgical instrumentactuation states. A sixth information structure 1016 associates surgeonskill levels during different activates with different messages. Aseventh information structure 1018 associates surgeon skill levels withduring different activates with different surgical instrument safetyactuation states. A ninth information structure 1019 associates recordedvideo information from individual surgeries with corresponding surgicalinstrument actuator state information and recorded surgeon eye movementsduring the surgeries in accordance with some embodiments.

In some embodiments, information in the various information structures1004-1019 are evaluated to identify correlations between surgeon skilllevels and surgical procedure results/risks. In some embodiments,information in the various information structures 1004-1019 areevaluated to identify correlations between patient safety concerns/risksand surgical activities during a surgical procedure. In someembodiments, teleoperated surgical procedures are evaluated to identifycorrelations between patient safety concerns/risks and surgicalinstrument actuator state during a surgical activity as a function ofsurgeon skill level. In some embodiments the storage atlas 1002 includesa tenth information structures 1020 to provide a correlation betweensurgical outcomes/risks and surgical instrument actuator state during asurgical activity as a function of surgeon skill level. Theseevaluations may involve machine learning (ML) techniques, for example.

The storage atlas 1002 includes data concerning surgeries on priorpatients and the prior surgeons who performed the prior surgeries. Insome embodiments, the storage atlas 1002 includes video images ofsurgical scenes from prior surgeries and corresponding annotations suchas text and telestration tags 1022. In some embodiments, the storageatlas 1002 includes recordings 1024 of surgical instrument actuatorstates during the prior surgeons' performance of surgical activities inthe prior surgeries.

FIG. 11 is an illustrative drawing representing an example of theseventh information structure 1018 included within the atlas 1002 in thestorage device 1004, which associates recorded video information from anindividual surgery with corresponding surgical instrument actuator stateinformation in accordance with some embodiments. In one aspect, videorecording images of patient anatomy during a surgery, surgicalinstrument actuator states during the surgery, and surgeon eye movementduring the surgery are time stamped (t1, t2 . . . tn) so as to produce achronological record of times of occurrence of surgical activities uponpatient anatomy, to provide a corresponding chronological record oftimes of occurrence of surgical instrument actuator states and toprovide a corresponding chronological record of surgical eye movementduring a surgical procedure. Thus, time stamps recorded during asurgical procedure are used to temporally align video images anatomy,with surgical instrument actuator states and surgeon eye gaze.

During a surgery, a user may annotate the video recording and thesurgical instrument actuation state recording with metadata thatindicate corresponding surgical activity such as vessel sealing, sutureknot-tying or blunt tissue dissection, for example. Time annotation mayinclude one or more of or a combination of written notes tagged to videoinformation and/or surgical instrument actuation state information,coloring or highlighting (e.g., telestration) of images in the videorecordings, for example. The annotations may be time stamped for use totemporally align them with corresponding video recording information andcorresponding recorded surgical instrument state information.

During a teleoperated surgical procedure, a surgical activity such asneurovascular bundle dissection (nerve sparing), often involves use ofmultiple surgical instruments such as a prograsper and robotic scissors,each having its own actuator state. Thus, different surgical activitiesoften require combinations of multiple surgical instrument skills. Forexample, performance of a continuous suturing surgical activity oftenrequires the following combination of surgical skills: instrument wristmanipulation, needle grasping, needle passing and orientation betweentwo instruments, tissue grasping, and needle driving.

FIGS. 12A-12C are illustrative drawings showing an example surgicalinstrument 1202 and an actuator assembly 1203 in which the surgicalinstrument is shown in three different example operational states inaccordance with some embodiments. The example instrument 1202 includes ajaw end effector 1204 that can transition between open and closed statesand a continuum of partially opened/partially closed states in between.The example instrument 1202 also includes a two degree of freedom(2-dot) wrist 1206 that can move between different two-dimensional (x,y) positional states. The example actuator assembly 1203 includes afirst actuator 1208, which in some embodiments includes a jaw motor (JM)used to actuate the jaw end effector 1204. The example actuator assembly1203 includes a second actuator 1210, which in some embodiments includesa wrist motor (WM) used to actuate the wrist 1206. During a surgery, thesurgical instrument 1202 may transition through multiple actuationstates corresponding to different activities during a surgicalprocedure. As represented in FIG. 12A, for example, a surgical proceduremay involve a first surgical activity in which the first actuator 1208(the JM) disposes the jaw end effector 1204 to a frilly open state andthe second actuator 1210 the (WM) disposes the wrist 1206 to a firstpositional state (x1, y1). As represented in FIG. 12B, for example, thesurgical procedure may involve a second surgical activity in which thefirst actuator 1208 transitions the jaw end effector 1204 to a fullyclosed state and the second actuator 1210 transitions the wrist 1206 toa second positional state (x2, y2). As represented in FIG. 12C, forexample, the surgical procedure may involve a third surgical activity inwhich the first actuator 1208 disposes the jaw end effector 1104 in apartially open/partially closed state and the second actuator 1210transitions the wrist 1206 to a third positional state (x3, y3).

In some embodiments, performance of a teleoperated surgical system inresponse to a surgeon's input control commands is scaled based upon thesurgeon's skill level for surgical activities performed using the systemduring the surgical procedure. More particularly, rate of movement of asurgical instrument in instrument space in response to user input at auser input command device in user space is scaled based upon at least inpart upon user skill level. For example, a record of the surgeon's skilllevel may indicate a novice skill level in performance of a needledriving surgical activity using a Large Suture Cut Needle Driversurgical instrument. In accordance with some embodiments, during theperformance of the needle driving surgical activity, the instrumentactuator is operated in a first (novice) mode in which the Large SutureCut Needle Driver automatically moves very deliberately (1:0.333scaling) relative to the user intent when near the respective tissue.Thus, the processor 58 is configured for a novice user of the LargeSuture Cut Needle Driver surgical instrument in which translation ofuser input movement to instrument movement is scaled to slow instrumentmovement. A one-unit movement in user space imparted at control inputs36, 38 is kinematically translated to a 0.333 unit movement of theinstrument in surgical instrument space. Alternatively, for example, arecord of the surgeon's skill level may indicate an intermediate skilllevel in performance of the needle driving surgical activity using aLarge Needle Driver surgical instrument. In accordance with someembodiments, during the performance of the needle driving surgicalactivity, the instrument actuator is operated in a second (intermediate)mode in which the Large Needle Driver moves deliberately in a 1:0.5scale relative to the user intent when near the respective tissue. Thus,the processor 58 is configured for intermediate skill level user of theLarge Needle Driver surgical instrument in which translation of userinput movement to instrument movement is scaled to slow instrumentmovement. A one-unit movement in user space imparted at control inputs36, 38 is kinematically translated to a 0.5 unit movement of theinstrument in surgical instrument space. As yet another alternative, forexample, a record of the surgeon's skill level may indicate anexperienced skill level in performance of the needle driving surgicalactivity using the Mega Needle Driver surgical instrument. In accordancewith some embodiments, during the performance of the needle drivingsurgical activity, the instrument actuator is operated in a third(experienced) mode in which the Mega Needle Driver moves in the samespeed, 1:1 scale, relative to the user intent when near the respectivetissue. Thus, the processor 58 is configured for an experienced skilllevel user of the Mega Needle Driver surgical instrument in whichtranslation of user input movement to instrument movement is scaled tomatch instrument movement. A one-unit movement in user space imparted atcontrol inputs 36, 38 is kinematically translated to a 1.0-unit movementof the instrument in surgical instrument space.

FIG. 13 is an illustrative drawing representing an example tenthinformation structure 1020 of the atlas 1002 stored in the computerreadable storage device 1004 that corresponds to an example surgicalprocedure to be performed by an example surgeon. The informationstructure 1020 associates surgical activities during the surgicalprocedure with surgeon skills. The information structure 1020 associatessurgeon skills with a surgeon's skill levels. The information structure1020 associates surgeon activities with surgical instrument actuatorstates. The information structure 1020 associates surgical state,surgical skill, surgical skill level, actuator state tuples withsurgical actuator safety sates.

A first column of the information structure 1020 indicates a list ofsurgical activities, A1, A2 and A3 to be performed during the examplesurgical procedure. A second column of the information structure 1020indicates lists of surgical skills required during each of theactivities and corresponding skill levels of the surgeon performing thesurgery for each of the skills. Specifically, in the example, activityA1 is associated with surgical skill S1, skill S3 and skill S5, and thesurgeon possess an experienced skill level LE for all three skills S1,S2 and S5. Activity A2 is associated with surgical skills S1, S2, S3,and the surgeon possess an experienced skill level La for skills S1 andS3 and S5 and possess a novice skill level L_(N) for skill S2. ActivityA3 is associated with surgical skills S1, S4, S6, and the surgeonpossess an experienced skill level L_(E) for skills S1 and S6 andpossess an intermediate skill level L₁ for skill S4. A third column ofthe information structure 1020 indicates surgical instrument actuationstates indicative of the occurrence of the surgical activities. Forexample, surgical instrument actuator state SIA_(A1) is indicative ofoccurrence of surgical state A1. Surgical instrument actuator stateSIA_(A2) is indicative of occurrence of surgical state A2. Surgicalinstrument actuator state SIA_(A3) is indicative of occurrence ofsurgical state A3. A fourth column of the information structure 1020indicates messages to be presented to a surgical team at differentstages of a surgical procedure, based upon surgical activity states. Forexample, surgical activity state A1 is associated with message, M_(A1E),directed to an experienced skill level surgeon; surgical activity stateA2 is associated with message, M_(A2N), directed to a novice skill levelsurgeon; and surgical activity state A3 is associated with message,M_(A3I), directed to an intermediate skill level surgeon.

A fifth column of the information structure 1020 indicates surgicalinstrument actuator safety states to be used during different surgicalactivities of the surgical procedure. For example, surgical activity A1is associated with an instrument actuator safety state SAI_(A1), whichindicates that the surgeon has an experienced skill level for surgicalactivity A1. Surgical activity A2 is associated with an instrumentactuator safety state SIA_(A2N), which indicates that the surgeon has anovice skill level for surgical activity A2. Surgical activity A3 isassociated with an instrument actuator safety state SIA_(A3I), whichindicates that the surgeon has intermediate skill level for surgicalactivity A3. It is noted that in this example, the associated instrumentactuator safety state is at a level of the lowest corresponding surgeonskill level applicable for the surgical activity. For example, foractivity A1, the surgeon's skill level is level L_(E) (experienced) forall three skills S1, S3, S5, and therefore, the instrument actuatorsafety state is SAI_(A1E) which corresponds to the experienced level.For activity A2, the surgeon's lowest skill level is level L_(N)(novice) for skill S2, and therefore, the instrument actuator safetystate is SAI_(A21N), which corresponds to the novice level. For activityA3, the surgeon's lowest skill level is level Li (intermediate) forskill S4, and therefore, the instrument actuator safety state isSAI_(A3I), which corresponds to the intermediate level.

Referring to the first row of the example information 1020 structure ofFIG. 13, for example, during a surgical procedure involving, activity A1of the surgery may involve tissue dissection, which requires skillS1=instrument wrist manipulation and orientation, skill S3=tissuegrasping and manipulation, and skill S5=cutting. Message MA1 _(E) mayindicate “to make small smooth cuts”. Surgical instrument safetyactuation state A1 _(E) may involve instrument is set to 1:1 scaling.

Referring to the second row of the example information 1020 structure ofFIG. 13, for example, during the surgical procedure involving, activityA 2 of the surgery may involve tissue suturing, which requires skill S1,skill S2=needle driving and skill S3. Message MA_(2N) may indicate “tobe careful with needle handling to avoid unnecessary tissue needlepiercings”. Surgical instrument safety actuation state A2 _(N) mayinvolve instrument is set to 1:0.333 scaling.

Referring to the third row of the example information 1020 structure ofFIG. 13, for example, during the surgical procedure involving, activityA3 of the surgery may involve suture knot-tying, which requires skillS1, skill S4=suture handling and skill S6=knot-tying. Message MA3 _(I)may indicate “to be careful managing length of suture tail”. Surgicalinstrument safety actuation state A3 _(I) may involve instrument is setto 1:0.5 scaling.

FIG. 14 is an illustrative flow diagram 1402 representing configurationof processor 58 to scale kinematic translation of user-to-instrumentmovement according to a surgical instrument safety actuation state basedat least in part upon surgeon skill level information in accordance withsome embodiments. Computer program code is used in some embodiments toconfigure one or more CPUs of the processor 58 to perform the process1402. In block 1404, a surgeon identification is received at an input toa computer processing system associated with the electronics cart 56. Inblock 1406, an identification of a surgical procedure is received at aninput of the computer processing system associated with the electronicscart 56. In block 1408, information included within the atlas 1002within information structures 1010, 1012, 1014, 1016 and 1018 is used toproduce an instance of the tenth information structure 1020 of FIG. 13relating to the identified surgeon and to the identified surgicalprocedure.

During the performance of the identified surgical procedure, block 1410tracks for each of one or more surgical instruments 26, which instrumentis mounted in at a surgical instrument manipulator 512 and also tracksoperational state of a surgical instrument actuator to determine, basedupon the surgical instrument actuator state information within the thirdcolumn of the produced instance of the tenth information structure 1020,when the surgical procedure is transitioning to an activity identifiedin the first column of the information structure 1020. In decision block1412, a determination is made as to whether at least one of aninstrument 26 mounted at a manipulator 512 a current instrument actuatorstate matches an actuator state associated with an activity. In responseto no match, control loops back to block 1410 and tracking continues. Inresponse to a match, block 1414 uses transitions the surgical instrumentactuator to an instrument actuator safety state identified in the fifthcolumn of the information structure 1020 that is associated with atleast one of an instrument 26 mounted at a manipulator 512 and currentinstrument actuator state and an associated surgeon skill levelindicated the second column of the information structure 1020. Block1416 configures the display device 32, 34 and/or 60 to present to asurgical team a safety message associated with the current instrumentactuator state and an associated surgeon skill level In someembodiments, control next flows back to block 1410, which continues totrack surgical instrument actuator state based upon other identifiedactuator state transition information, for example.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. For example, in some embodiments, a virtual surgical systemcomprises a first virtual robotic surgical instrument. The systemincludes an image generation device to produce images of virtualanatomical tissue and of the first virtual robotic surgical instrument.A user display is coupled to the image generation device so as to showto a user, the generated images of the virtual anatomical tissue and ofthe first virtual robotic surgical instrument. A user input commanddevice is coupled to receive user input commands to control virtualmovement of the first virtual robotic surgical instrument. A virtualmovement controller is coupled to receive the user input commands fromthe input command device and configured to control virtual movement ofthe first virtual robotic surgical instrument in response to the userinput commands and to scale a rate of virtual movement of the firstvirtual robotic surgical instrument, based at least in part upon asurgical skill level at using the first virtual robotic surgicalinstrument of the user providing the received user input commands, froma rate of movement indicated by the user input commands received at theuser input command device.

Moreover, in some embodiments, a method is provided to operate a virtualsurgical system that includes a virtual robotic surgical instrumentmanipulator. The method includes receiving user input commands from auser to control movement of a virtual robotic surgical instrumentmounted at the virtual robotic surgical instrument manipulator. Themethod further includes determining an identification of a virtualrobotic surgical instrument mounted at the virtual robotic surgicalinstrument manipulator during the receiving the user input commands. Themethod also includes scaling a rate of movement of the virtual roboticsurgical instrument, based at least in part upon a skill level of theuser at use of the identified virtual surgical instrument, from a rateof movement indicated by the user input commands. One of ordinary skillin the art would recognize many variations, alternatives, andmodifications. Thus, the scope of the disclosure should be limited onlyby the following claims, and it is appropriate that the claims beconstrued broadly and in a manner consistent with the scope of theembodiments disclosed herein.

1. A teleoperated surgical system comprising: a first robotic surgicalinstrument; an image capture device orientable toward a surgical sitefor capturing images of anatomical tissue and of first robotic surgicalinstrument; a user display coupled to the image capture device so as toshow to a user, the captured images of the anatomical tissue and of thefirst robotic surgical instrument; a user input command device coupledto receive user input commands to control movement of the first roboticsurgical instrument; and a movement controller coupled to receive theuser input commands from the input command device and configured tocontrol movement of the first robotic surgical instrument in response tothe user input commands and to adjust a rate of movement of the firstrobotic surgical instrument, based at least in part upon a surgicalskill level at using the first robotic surgical instrument of the userproviding the received user input commands, based on movement of theuser input command device during the user input commands.
 2. Theteleoperated surgical system of claim 1, wherein the movement controlleris configured to adjust a rate of movement of the first robotic surgicalinstrument to a slower rate of movement, based at least in part upon alower skill level of the user at using the first robotic surgicalinstrument, based on movement of the user input command device duringthe user input commands.
 3. The teleoperated surgical system of claim 1,wherein the movement controller is configured to adjust a rate ofmovement of the first robotic surgical instrument to a slower, firstrate of based at least in part upon a lower skill level of the user atusing the first robotic surgical instrument, based on movement of theuser input command device during the user input commands; and whereinthe movement controller is configured to adjust a rate of movement ofthe first robotic surgical instrument to a faster, second rate ofmovement, based at least in part upon a higher skill level of the userat using the first robotic surgical instrument, based on movement of theuser input command device during the user input commands.
 4. Theteleoperated surgical system of claim 3, wherein the second rate ofmovement matches a rate of movement indicated by the user input commandsreceived at the user input command device.
 5. The teleoperated surgicalsystem of claim 3, wherein the second rate of movement is slower than arate of movement indicated the user input commands received at the userinput command device.
 6. The teleoperated surgical system of claim 1,wherein the movement controller is configurable to scale a rate ofmovement of the first robotic surgical instrument to a first rate ofmovement, based at least in part upon a first skill level of the user atusing the first robotic surgical instrument, from a rate of movementindicated by the user input commands received at the user input commanddevice; wherein the movement controller is configurable to scale a rateof movement of the first robotic surgical instrument to a second rate ofmovement, based at least in part upon a second skill level of the userat using the first robotic surgical instrument, from a rate of movementindicated by the user input commands received at the user input commanddevice; and wherein the movement controller is configurable to scale arate of movement of the first robotic surgical instrument to a thirdrate of movement, based at least in part upon a third skill level of theuser at using the first robotic surgical instrument, from a rate ofmovement indicated by the user input commands received at the user inputcommand device.
 7. The teleoperated surgical system of claim 1, whereinthe user display is configured to display a message in response to theuser input commands based at least in part upon a skill level of theuser providing the received user input commands received at the userinput command device.
 8. The teleoperated surgical system of claim 1,wherein the user display is configured to display a first message inresponse to the user input commands based at least in part upon a firstskill level of the user providing the received user input commandsreceived at the user input command device; wherein the movementcontroller is configurable to adjust scale a rate of movement of thefirst robotic surgical instrument to a first rate of movement, based atleast in part upon a first skill level of the user at using the firstrobotic surgical instrument, based on movement of the user input commanddevice during the user input commands; wherein the user display isconfigured to display a second message in response to the user inputcommands based at least in part upon a second skill level of the user atusing the second robotic surgical instrument; and wherein the movementcontroller is configurable to adjust a rate of movement of the firstrobotic surgical instrument to a second rate of movement, based at leastin part upon a second skill level of the user at using the first roboticsurgical instrument, based on movement of the user input command deviceduring the user input commands.
 9. The teleoperated surgical system ofclaim 1, wherein the image capture device includes a stereoscopicdisplay.
 10. The teleoperated surgical system of claim 1, wherein theuser input command device includes hand grips.
 11. A teleoperatedsurgical system comprising: a robotic surgical instrument manipulatorconfigured to selectably mount at least one of multiple robotic surgicalinstruments; an image capture device orientable toward a surgical sitefor capturing images of anatomical tissue and of the at least onerobotic surgical instrument; a user display coupled to the image capturedevice so as to show to a user, the captured images of the anatomicaltissue and of the at least one robotic surgical instrument; a user inputcommand device coupled to receive user input commands to controlmovement of the at least one robotic surgical instrument; and a movementcontroller coupled to receive the user input commands from the inputcommand device and configured to control movement of the at least onerobotic surgical instrument in response to the user input commands andto adjust a rate of movement of the at least one robotic surgicalinstrument, based at least in part upon a surgical skill level at usingthe first robotic surgical instrument of the user providing the receiveduser input commands, based on movement of the user input command deviceduring the user input commands.
 12. The teleoperated surgical system ofclaim 11, wherein the multiple robotic instruments include a selectablymountable first robotic surgical instrument and a selectably mountablesecond robotic surgical instrument; wherein the movement controller isconfigured to control movement of the first robotic surgical instrument,when selectably mounted on the robotic surgical instrument manipulator,in response to the user input commands and to adjust to a first rate ofmovement of the first robotic surgical instrument, based at least inpart upon a surgical skill level at using the first robotic surgicalinstrument of the user providing the received user input commands, basedon movement of the user input command device during the user inputcommands to the first robotic surgical instrument; and wherein themovement controller is configured to control movement of the secondrobotic surgical instrument, when selectably mounted on the roboticsurgical instrument manipulator, in response to the user input commandsand to adjust to a second rate of movement of movement of the secondrobotic surgical instrument, based at least in part upon a surgicalskill level at using the second robotic surgical instrument of the userproviding the received user input commands, based on movement of theuser input command device during the user input commands to the secondrobotic surgical instrument.
 13. The teleoperated surgical system ofclaim 11, wherein the user display is configured to display a one ofmultiple respective messages in response to the user input commands,based at least in part upon a skill level of the user at using the atleast one robotic surgical instrument mounted on the robotic surgicalinstrument manipulator.
 14. A method to operate a teleoperated surgicalsystem that includes a robotic surgical instrument manipulator,comprising: receiving user input commands from a user to controlmovement of a robotic surgical instrument mounted at the roboticsurgical instrument manipulator; determining an identification of arobotic surgical instrument mounted at the robotic surgical instrumentmanipulator during the receiving the user input commands; and adjustinga rate of movement of the robotic surgical instrument, based at least inpart upon a skill level of the user at use of the identified surgicalinstrument, based on the user input commands.
 15. The method of claim 14further including: receiving an identification of a user; receiving anindication of skill of the identified user at use of the identifiedsurgical instrument.
 16. The method of claim 14 further including:wherein adjusting the rate of movement of the robotic surgicalinstrument includes, adjusting to a first rate of movement from a rateof movement indicated by the user input commands, in response to theidentified surgical instrument including a first surgical instrument andthe user having a first skill level at use of the first surgicalinstrument; adjusting to a second rate of movement from a rate ofmovement indicated by the user input commands, in response to theidentified surgical instrument including a second surgical instrumentand the user having a second skill level at use of the second surgicalinstrument.
 17. The method of claim 16 further including: receiving anidentification of a user; receiving an indication of skill of theidentified user at use of the first surgical instrument; and receivingan indication of skill of the identified user at use of the secondsurgical instrument.
 18. The method of claim 16 further including:receiving an identification of a user; receiving an indication of lowerskill level of the identified user at use of the first surgicalinstrument; and receiving an indication of a higher skill of theidentified user at use of the second surgical instrument; whereinadjusting the first rate of movement includes adjusting to a slower rateof movement from the rate of movement indicated by the user inputcommands.
 19. The method of claim 18, wherein adjusting to the secondrate of movement includes adjusting to a rate of movement that matches arate of movement indicated by the user input commands.
 20. The method ofclaim 18, wherein adjusting to the second rate of movement includesadjusting to a slower rate of movement from the rate of movementindicated by the user input commands. 21-22. (canceled)